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Background / Context: 

Effective school reform requires communication with and buy-in from key stakeholders. In 
recent years, stakeholder surveys have become increasingly prominent tools for promoting better 
understanding of how school factors contribute to student success. Specifically, surveys are 
employed by researchers, practitioners, and policy makers to inform policy development and 
improvement, to evaluate the effectiveness of reform (e.g., U.S. Department of Education, 1998), 
to examine the prevalence of effective practices (e.g., Garet, Porter, Birman, Desimone, & Yoon. 
2002), to make international comparisons (e.g., Schmidt, McKnight, & Raizen, 2001), and to 
compare conditions of teaching and teachers across districts and states (Desimone, Smith, & 
Frisvold, 2010). Survey data are also commonly used to measure implementation in large-scale 
studies of state and local efforts such as comprehensive school reform (e.g., Ross et al., 1997). 

Research suggests that thoughtful, well-designed surveys are an effective tool for gaining key 
insights into school climate, culture, and practices (Desimone and Le Floch, 2004). Both the 
National Association of Secondary School Principals (NASSP) and National Education 
Association (NEA) recommend regular use of stakeholder surveys to promote healthy learning 
environments for students (NASSP, 2014; NEA, 2012). Considering the prominent role of 
surveys in educational research and policy decisions, scholars and practitioners have placed a 
strong emphasis on assessing and increasing the reliability and validity of survey data. 

The current study is based on a collaboration between McREL International (McREL) and the 
Victorian Department of Education and Early Childhood Development (DEECD) to incorporate 
current research on what matters most to student success into a next generation school survey for 
the educators of Victoria. The intent of the survey is to provide school leaders state-wide with 
intuitive, practical guidance that they can immediately apply to improve their own leadership 
behaviors, school climate, teaching, and ultimately, student learning. Through a process of 
rigorous field testing and psychometric analysis, McREL created the new School Staff Opinion 
Survey with the intent to meet the highest possible standards of validity and reliability. To 
produce an essential tool for improving school and student performance, survey items were 
linked to behaviors and factors than have been related through research to student engagement 
and learning. 

Purpose / Objective / Research Question / Focus of Study: 

The key purpose of this study was to investigate the relation between Victorian school scores on 
the School Staff Opinion Survey and student achievement as measured by The National 
Assessment Program - Literacy and Numeracy (NAPLAN). As a matter of background, during 
2013, researchers at McREL developed a new Staff Opinion Survey at the request of DEECD 
with modules in five vital areas of school functioning: 1) School Climate, 2) School Leadership 
(the leadership of teams), 3) Professional Development, 4) Staff Safety and Well-being, and 
Teaching and Learning 

The goal of the survey development project, which was completed in 2014, was to measure 
multiple constructs with research suggesting their linkages to valued outcomes in each of the 
areas above and, where possible (i.e., supported by psychometric evidence), to combine the 
construct measures within each module to produce a single overall module Index score. 
Psychometric analyses supported the construction of an Index score combining each of the 
modules’ scores except for Teaching and Learning. For this reason, the items were combined to 
create overall Index scores for School Climate, School Leadership, Professional Learning, and 
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Staff Safety and Well-being but not for Teaching and Learning 1 . Thus, the analyses reported here 
employ index scores where available. 

The main research question for this predictive validity study was: How do various measures 
included in the Victorian new Staff Opinion Survey relate to one another and to student 
achievement outcomes on the NAPLAN assessment? 

Setting: 

Surveys were administered by McREL via an online survey platform to staff members in 64 
schools throughout the state of Victoria. Student NAPLAN data were collected by the DEECD 
from participating schools and delivered to McREL for merging with school-level staff survey 
data. 

Population / Participants / Subjects: 

To conduct this study with sufficient representation of the Victorian primary schools, McREL 
researchers purposefully over-sampled primary schools during the trial of the new Staff Opinion 
Survey. As a result, the data for this study are comprised of a stratified random sample of 62 
primary schools in Victoria that are representative of the population of primary schools in 
Victoria in terms of size and socioeconomics. 

Intervention / Program / Practice: 

To provide Victorian educators with insights and how-to guidance they can apply to focus 
continuous school improvement efforts, the items and subscales for each survey module were 
refined through triangulation with extant research, expert review, field tests with Victorian staff, 
and rigorous statistical analysis of items trialed in the pilot sample of Victorian schools. As a 
result, McREL produced a framework with empirical and theoretical support linking the survey 
modules to one another and to student outcomes, a design for school reports, and a How-to- 
Guide to help schools plan improvement based on their individual survey data. 

Research Design: 

Most of the constructs found in the new Staff Opinion Survey were developed through extensive 
item- writing, cognitive interview, and expert panel review processes. Several of the constructs 
had been validated in extant research that has examined their relations to one another and student 
outcomes. In particular, Goddard, Goddard, Kim, and Miller (in press) recently showed that all 
of the direct and indirect relations connecting instructional leadership to student outcomes 
through teacher collaboration and collective efficacy are positive and statistically significant. In 
addition, there has been extensive research on the measures of Trust in Students and Parents 
(Goddard, Tschannen-Moran, & Hoy, 2001; Goddard, Salloum, & Berebitsky, 2009), and 
academic emphasis (e.g., Goddard, Sweetland, & Hoy, 2000) linking both to student outcomes. 
We leveraged this extant support by including these validated measures in the School Leadership 
and School Climate modules and by examining the correlations among the Index Scores for the 
School Climate, School Leadership, Professional Learning, and Staff Safety and Well-being 
modules. We also examined the degree to which the Index scores are related to student 
NAPLAN scores. 

Data Collection and Analysis: 


1 Results from the Teaching and Learning module are omitted from this presentation due to the lack of psychometric 
support for the construction of an Index score. Results from individual Teaching and Learning subscales are 
discussed elsewhere (McREL International, 2015). 
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Data describing the participating trial schools and the students they served were obtained from 
the Victorian DEECD. These included the numeracy and literacy NAPLAN scores of grade 5 
students in 2014 and the 2012 NAPLAN grade 3 scores for the same students, where available. 
The Staff Opinion Survey module data in the areas of School Climate, School Leadership, 
Professional Learning, and Staff Safety and Well-being were provided by McREL, which 
administered the trial survey data collection. The psychometric properties of the scores used in 
these analyses were generally strong and have been reported extensively elsewhere (McREL 
International, 2014). 

The main analyses were multilevel models that tested the predictive validity of all module index 
scores (climate, leadership, professional learning, and staff safety and well-being). The general 
form of the hypotheses we tested was: “What is the relationship between the construct measured 
and differences among Victorian Primary schools in student outcomes?” (i.e., X->Y). Multilevel 
modeling was chosen as the primary analytic method because it properly estimates standard 
errors and avoids the problems of aggregation bias associated with single-level analyses. 2014 
(grade 5) student NAPLAN scores in numeracy and literacy served as the outcome variables. 
2012 (grade 3) student NAPLAN scores in numeracy and literacy served as statistical controls 
for prior achievement in the assessed content areas and ensured that our analyses examine 
learning. We also included student-level demographic variables for gender, indigenous status, 
cultural or linguistic background other than English, parent education, and parent occupation. At 
the school level, we included a socioeconomic status indicator and a measure of school size, 
however, the school size indicator was dropped because it was unrelated to student outcomes. 

We also tested several mediated relationships that are central to the Department’s ongoing school 
improvement efforts. Lor example, the research on education leadership demonstrates clearly 
that because principals do not teach, their impact on student outcomes is largely indirect, flowing 
through their impact on school climate, the professional development they provide teachers, and 
so on. Our own research also indicates that while there is not a strong direct statistical link 
between measures of leadership and student outcomes, leadership is related to student outcomes 
through important school climate variables. Therefore, we tested several plausible indirect, or 
mediated, relationships. Similar to the predictive validity tests above, we employed multilevel 
structural equation modeling (ML-SEM) to test these relationships. As with multilevel modeling, 
ML-SEM allows the simultaneous estimation of a within-schools model to address student 
outcomes and a between- schools model to estimate the impact of school-level factors, 
particularly Staff Opinion Survey measures, on differences among schools in student outcomes. 
An example mediated model of the type we tested is depicted graphically in Ligure 1. 

Findings / Results: 

Student NAPLAN literacy and numeracy scores collected in the same year were correlated 
relatively low (only 43% statistical overlap). This means that a variable that is a significant 
predictor of one academic outcome is not as likely to predict the other as strongly. This 
manifested in larger effects (on average) for the Staff Opinion Survey scores on literacy than on 
numeracy. That said, the direction of the effects of the Staff Opinion Survey measures was 
consistently positive for both content areas. 

Correlational analyses showed that the Index Scores for School Leadership, School Climate, 
Professional Learning, and Staff Safety and Well-being were highly interrelated. In other words, 
schools that scored high in one area tended to score high in the other areas. Lurthermore, Grade 5 
(2014) numeracy and literacy NAPLAN scores collected after the administration of the Staff 
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Opinion Survey trial were positively and significantly correlated with Index scores for School 
Climate, School Leadership, and Staff Safety and Well-being. The Professional Learning Index 
Score was significantly and positively related to Grade 5 (2014) numeracy outcomes but not 
literacy. The module Index score most strongly related to differences among schools in student 
outcomes was School Climate (see Table 2). 

The subscales in the School Climate module most strongly related to student outcomes were 
collective efficacy, trust in students and parents, academic emphasis, and parent and community 
involvement. These constructs combined to represent a construct called academic optimism 
which permeates the school and reaches into the community. Academic optimism was predicted 
strongly by the degree to which staff reported collaborating on instructional improvement. In 
addition, the degree to which staff reported collaborating on instructional improvement was 
strongly predicted by the strength of their school leadership team’s instructional leadership. 

The findings of the most complex of our tests yielded a promising model that replicated the 
findings of Goddard, Goddard, Kim and Miller (in press) and explained over 75% of the variance 
among schools in student outcomes as follows: 

School Leadership -> Teacher Collaboration -> Academic Optimism -> Year 5 NAPLAN 
Literacy. 

The results of the model above suggests that the schools with the greatest levels of collaboration 
for instructional improvement were those in which teachers reported strong school leadership on 
the part of the school leadership team (see Table 3). In turn, these were the most academically 
optimistic schools and the ones with systematically higher levels of year 5 literacy outcomes. 

Conclusions: 

The findings of the predictive validity study indicate that the School Climate, School Leadership, 
Professional Learning, and Staff-safety and Well-being scores were highly interrelated and 
positively correlated with student NAPLAN outcomes. A key finding is that the academic 
optimism measure (collective efficacy, trust in students and parents, academic emphasis and, 
parent and community involvement) was the strongest predictor of literacy outcomes and that 
academic optimism tended to be greater in schools where staff reported collaborating on 
instructional improvement and professional development. In addition, such staff collaboration 
was more likely where school leadership was stronger. This suggests that instructional leadership 
and professional collaboration are possible pathways to improving instruction and student 
learning. Thus, the Staff Opinion Survey appears to have succeeded in measuring aspects of 
School Leadership, School Climate, Professional Learning, and Staff Safety and Well-being that 
are positively interrelated and that can be used to predict literacy outcomes in the primary 
schools of Victoria. The results of this study will help guide future initiatives set forth by the 
DEECD to leverage school-level predictors of student achievement throughout the state of 
Victoria. In addition, the results of the mediated models replicate research conducted in the U.S. 
(Goddard, Goddard, Kim and Miller, in press) and therefore have broad implications for 
educational policy and school practice world- wide. 
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Appendix B. Tables/Figures 


Table 1. Primary Schools Trial Sample Design 


Group 

SES 

School 

Size 

Number 
of eligible 
schools 

Number of 
schools 
sampled 

Percentage 
of schools 
sampled 

1 

High 

Large 

310 

16 

5% 

2 

High 

Small 

310 

16 

5% 

3 

Low 

Large 

293 

16 

5% 

4 

Low 

Small 

292 

16 

5% 

Totals 

1,205 

64 

5% 


School Teacher I Academic H Student 

Leadership Collaboration Optimism Outcomes 


Figure 1 . Mediated model tested in the current study 
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Table 2. Correlations among Survey Module Index Scores for School Climate, Leadership, Professional Learning 
and Staff Safety/Wellbeing and Numeracy and Literacy Baseline (Grade 3 2012) and Outcome (Grade 5 2014) 
Scores 






Staff Safety 

Grade 3 

Grade 3 

Grade 5 

Grade 5 




Professional 

and 

(2012) 

(2012) 

(2014) 

(2014) 



Leadership 

Learning 

Wellbeing 

Numeracy 

Literacy 

Numeracy 

Literacy 



Index 

Index 

Index 

Score 

Score 

Score 

Score 

School 

Pearson Correlation 

.820” 

.682” 

.819” 

.114” 

.112” 

.151” 

.151” 

Climate 

Sig. (2 -tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

Index 

N 

2595 

2595 

2606 

2141 

2134 

2349 

2344 

Leadership 

Pearson Correlation 


.849” 

.903” 

.077” 

.046* 

.113” 

.070” 

Index 

Sig. (2 -tailed) 


.000 

.000 

.000 

.036 

.000 

.001 


N 


2595 

2595 

2131 

2123 

2338 

2333 

Professional 

Pearson Correlation 



.745” 

.017 

.001 

.050’ 

.012 

Learning 

Sig. (2 -tailed) 



.000 

.421 

.948 

.016 

.562 

Index 

N 



2595 

2131 

2123 

2338 

2333 

Staff Safety 

Pearson Correlation 




.103” 

.072” 

.126” 

.094” 

and 

Sig. (2 -tailed) 




.000 

.001 

.000 

.000 

Wellbeing 

Index 

N 




2141 

2134 

2349 

2344 

Grade 3 

Pearson Correlation 





.688** 

.748*’ 

.629” 

(2012) 

Sig. (2 -tailed) 





.000 

.000 

.000 

Numeracy 

Score 

N 





2105 

2030 

2028 

Grade 3 

Pearson Correlation 






.593” 

.755” 

(2012) 

Sig. (2 -tailed) 






.000 

.000 

Literacy 

Score 

N 






2028 

2026 

Grade 5 

Pearson Correlation 







.656” 

(2014) 

Sig. (2 -tailed) 







.000 

Numeracy 

Score 

N 







2320 


Notes. *p <.05 ** p <.01; Baseline achievement is from Year 3, 2012 NAPLAN 
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Table 3. Mediated Model: Leadership -> Teacher Collaboration Academic Optimism 
Achievement Numeracy and Literacy Outcomes Model 



Leadership Teacher Collaboration A Academic Optimism -> 
Achievement Model 


Estimate 

S.E. 

p -value 

Estimate 

S.E. 

p - value 

Leadership -> Teacher Collaboration 

0.71 

0.07 

.000** 

... 

... 

... 

Teacher Collaboration -> Academic Optimism 

0.85 

0.12 

.000** 

... 

... 

... 

Leadership -> Teacher Collaboration -> 

Academic Optimism 

0.61 

0.10 

.000** 

... 

... 

... 

Academic Optimism -> Achievement 


Numeracy 



Literacy 


Between School 







High SES 

-0.52 

4.02 

.896 

8.82 

5.04 

.080 

Baseline Achievement 

0.79 

0.11 

.000** 

0.73 

0.11 

.000** 

Academic Optimism Index Score 

3.54 

2.72 

.194 

7.89 

2.29 

.000** 

/^-Square 

0.81 

0.12 

.000** 

0.72 

0.10 

.000** 

Indirect Effects: 







Teacher Collaboration -> Academic Optimism 
-> Achievement 

3.01 

2.34 

.199 

6.72 

2.12 

.002 

Leadership -> Teacher Collaboration -> 
Academic Optimism -> Achievement 

2.15 

1.64 

.191 

4.79 

1.62 

.003 


Notes. *p <.05 ** p <.01; Baseline achievement is from Year 3, 2012 NAPLAN 
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